Designing classifier fusion systems by genetic algorithms

We suggest two simple ways to use a genetic algorithm (GA) to design a multiple-classifier system. The first GA version selects disjoint feature subsets to be used by the individual classifiers, whereas the second version selects (possibly) overlapping feature subsets, and also the types of the individual classifiers. The two GAs have been tested with four real data sets: Heart, Satimage, Letters, and Forensic glasses (tenfold cross validation, except for Satimage where we used only two splits). We used three-classifier systems and basic types of individual classifiers (the linear and quadratic discriminant classifiers and the logistic classifier). The multiple-classifier systems designed with the two GAs were compared against classifiers using: 1) all features; 2) the best feature subset found by the sequential backward selection (SBS) method; and 3) the best feature subset found by a GA (individual classifier!). We found that: 1) the multiple-classifier system derived through the GA, Version 2, yielded the smallest training error rate in all experiments; and 2) with Satimage and Forensic glasses data, it also produced the smallest test error rate. Generalizing on the basis of these experiments is not straightforward because the differences between the error rates in the comparison appeared to be too small. GA design can be made less prone to overtraining by including penalty terms in the fitness function accounting for the number of features used.